Trading Spaces: How Humans and Humanoids use Speech and Gesture to Give Directions

Humans frequently accompany direction-giving with gestures. These gestures have been shown to have the same underlying conceptual structure as diagrams and direction-giving language, but the puzzle is how they communicate given that their form is not codified, and may in fact differ from one person to the next. Based on results from a study on language and gesture in direction-giving, we propose a framework to analyze such gestural images into semantic units (image description features), and to link these units to morphological features (hand shape, trajectory, etc.). This feature-based framework in turn allows us to implement an integrated microplanner for multimodal descriptions that derives the form of both natural language and gesture directly from communicative goals. In this way we have been able to realize an embodied conversational agent that can perform appropriate speech and novel gestures in direction-giving conversation with real humans.